Split unsplit dividend totals by SOI qualified share#202
Merged
Conversation
normalize_dividend_columns dumped a dividend total with no observed qualified/non-qualified breakdown (e.g. CPS DIV_VAL) 100% into non_qualified_dividend_income, zeroing qualified dividends on every CPS-native row and inverting the national split (qualified 18% vs the SOI ~78%; qualified_dividend_income -65% vs its $291B target). Allocate an unsplit total by the SOI qualified share (0.78, from PUF E00650/E00600) instead. PUF data is unaffected (it carries the split). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the qualified / non-qualified dividend split inversion in
normalize_dividend_columns.When a record carries only a dividend total (e.g. CPS
DIV_VAL, which reports no qualified/non-qualified breakdown) and no observed component split, the function dumped 100% of the total intonon_qualified_dividend_incomeand leftqualified_dividend_incomeat $0. Because this runs on the merged persons (pipelines/us.py:10854,11090) where every CPS-native dividend row has a total but no split, it zeroed qualified dividends on the entire CPS spine and inverted the national split.Measured on the collapse-scaffold RC (audit dashboard, scored via microdf weighted
.sum()against the SOI targets):qualified_dividend_income)qualified_dividend_incomewas −65% vs its $291B target; the total dividend was roughly right, so the error was purely in the split.Fix
Allocate an unsplit dividend total by the SOI qualified share (
UNSPLIT_DIVIDEND_QUALIFIED_SHARE = 0.78, basis: SOI 2015 PUF E00650/E00600 = $204.0B/$260.9B = 0.782) instead of defaulting the whole amount to non-qualified. Applies in the two branches that previously produced all-non-qualified output (has_qualified and has_non_qualifiedtotal-only rows, and the no-componentselsebranch).This mirrors how the incumbent eCPS splits CPS
DIV_VALvia a qualified fraction. The 0.78 flat share is a documented first-order constant; a future refinement is an income-conditional share.Blast radius
normalize_dividend_columnshas 4 call sites. PUF data is unaffected — it carries the E00650/E00600 split, so the total-only branch never fires there. Only merged-persons rows that have a dividend total but no observed split (the CPS-native spine) change.Tests
test_normalize_dividend_columns_coalesces_sparse_total_aliases_by_row(row 0 now splits the unsplit total 78/22 instead of 0/100; rows with observed components unchanged).test_normalize_dividend_columns_splits_unsplit_total_by_qualified_share.test_normalize_dividend_columns_prefers_atomic_components_over_totalsstill passes (observed components are preserved, never reallocated).Validation note
Unit-tested. Full national validation requires a data rebuild (the qualified/non-qualified split feeds calibration + tax computation); expected to move
qualified_dividend_incomefrom ~$102B toward the ~$291B SOI target and deflatenon_qualified_dividend_incomecorrespondingly. Part of the calibration-surface fixes tracked in #200 (this is the confirmed, self-contained one; the interest seam, wage/NIPA target profile, and tax-expenditure target handling remain there).